Apparatus, optical apparatus, image pickup method, and non-transitory computer-readable storage medium

ABSTRACT

An apparatus includes a sensor, at least one processor; and at least one memory coupled to the at least one processor, the memory having instructions that, when executed by the at least one processor, perform to acquire lens information related to a lens apparatus, acquire array information related to an optical apparatus, acquire aberration correction information based on the lens information and the array information, cut out a plurality of images in a tile form from an image signal output by the sensor picking up a light beam incident from an object through a filter of the optical apparatus, and correct an aberration in the plurality of images, based on the aberration correction information.

BACKGROUND Technical Field

The aspect of the embodiments relates to an image pickup apparatus forsimultaneously picking up a plurality of images using an opticalapparatus that is replaceable with respect to the image pickup apparatusas a multispectral camera, a control method for the image pickupapparatus, a storage medium, and an optical apparatus.

Description of the Related Art

Multispectral cameras, also referred to as multi-band cameras, have beenknown as cameras configured to capture a plurality of different spectralcomponents in a spectroscopic spectrum. Multispectral cameras areutilized for, for example, food inspection based on images each obtainedfrom a different spectral component.

An example of an image pickup system adopted in multispectral cameras isa tiled multispectral filter array having a configuration in which, on afront side (object side) of an image sensor, filters are arranged forrespective partial areas (tiles) in a manner corresponding to an imagepickup area of an image sensor. The adoption of this system makes itpossible to simultaneously acquire a plurality of tiled-form band images(tile images) from the image sensor in one image pickup operation. Forexample, Japanese Patent Application Laid-Open No. 2020-64164 discussesan optical apparatus that is configured to disperse a light beam intodifferent wavelength bands and includes a replaceable lens array and areplaceable band-pass filter array, and also discusses an image pickupsystem including the optical apparatus.

In images obtained by a digital camera, an image distortion or an imagesize difference between different band images occur due to an opticalapparatus, such as a lens. An image distortion caused by a lens and thelike is generally referred to as a distortion aberration, and an imagesize difference due to a lens and the like is generally referred to as amagnification chromatic aberration. These aberrations occur also when amultispectral camera is used. For example, Japanese Patent ApplicationLaid-Open No. 2019-020951 discusses a technique for correcting adistortion aberration and a magnification chromatic aberration for eachband image.

However, in the technique discussed in Japanese Patent ApplicationLaid-Open No. 2020-64164, an image distortion or an image sizedifference still may occur depending on a position of a band-pass filterin the band-pass filter array.

Further, Japanese Patent Application Laid-Open No. 2019-020951 isconsidered to have little details on a method for reducing an imagedistortion or an image size difference in accordance with the lens arrayand the band-pass filter array in the optical apparatus that isreplaceable with respect to a camera. Consequently, there may be apossibility that an image distortion due to optical characteristics ofan optical apparatus (which is, for example, an interchangeable lens andan adapter apparatus) that is replaceable with respect to a cameracannot be accurately corrected.

SUMMARY

According to an aspect of the disclosure, an apparatus includes asensor, at least one processor; and at least one memory coupled to theat least one processor, the at least one memory having instructionsthat, when executed by the at least one processor, perform to acquirelens information related to a lens apparatus, acquire array informationrelated to an optical apparatus, acquire aberration correctioninformation, based on the lens information and the array information,cut out a plurality of images in a tile form from an image signal outputby the sensor picking up a light beam incident from an object through afilter of the optical apparatus, and, correct an aberration in theplurality of images, based on the aberration correction information,wherein the filter includes a plurality of areas to disperse the lightbeam from the object into a plurality of spectral components, andwherein each of the plurality of images corresponds to a different areaamong the plurality of areas of the filter.

Further features of the disclosure will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a configuration example of animage pickup apparatus according to a first exemplary embodiment of thedisclosure.

FIGS. 2A to 2C are diagrams each illustrating an image pickup systemaccording to the first exemplary embodiment.

FIG. 3 is a flowchart illustrating aberration correction processingaccording to the first exemplary embodiment.

FIGS. 4A to 4D are diagrams each illustrating a method for cutting outtile images and performing aberration correction processing on the tileimages according to the first exemplary embodiment.

FIG. 5 is a flowchart illustrating density correction processingaccording to a second exemplary embodiment of the disclosure.

DESCRIPTION OF THE EMBODIMENTS (Basic Configuration of Image PickupApparatus 1)

Exemplary embodiments of the disclosure will be described below withreference to the accompanying drawings. FIG. 1 is a block diagramillustrating a configuration example of an image pickup apparatus 1according to a first exemplary embodiment of the disclosure.

One or more functional blocks illustrated in FIG. 1 may be implementedby hardware, such as an application-specific integrated circuit (ASIC)or a programmable logic array (PLA), or may be implemented by causing aprogrammable processor, such as a central processing unit (CPU) or amicro processing unit (MPU), to execute software.

The one or more functional blocks can also be implemented by acombination of software and hardware.

Accordingly, even if different functional blocks are described asoperating entities in the following description, these functional blockscan be implemented by the same hardware entity.

As illustrated in FIG. 1 , the image pickup apparatus 1 according to thepresent exemplary embodiment is a lens-interchangeable image pickupapparatus to which an optical apparatus 2 and a lens apparatus 3 aredetachably attachable via respective mount portions (not illustrated).However, the configuration of the image pickup apparatus 1 is notlimited to this example. For example, the image pickup apparatus 1 caninclude the optical apparatus 2 and the lens apparatus 3. The opticalapparatus 2 and the lens apparatus 3 can be integrated together. A basicconfiguration of the image pickup apparatus 1 is described below usingan example case where the optical apparatus 2 and the lens apparatus 3are attached to the image pickup apparatus 1. The optical apparatus 2and the lens apparatus 3 will be described in detail below.

The image pickup apparatus 1 includes an image pickup unit 110, amicrocomputer 120, an operation unit 130, a display unit 140, a storageunit 150, a volatile memory 160, a non-volatile memory 170, a correctioninformation acquisition unit 180, and an image processing unit 190.

The image pickup unit 110 is a charge accumulation type solid-stateimage sensor, such as a charge-coupled device (CCD) sensor or acomplementary metal-oxide semiconductor (CMOS) sensor, that receives anoptical image formed by light passed through the lens apparatus 3 anddispersed by the optical apparatus 2. Information related to electriccharges obtained by photoelectric conversion (image pickup) performed ona light beam which is from an object and is obtained via the lensapparatus 3 is subjected to analog-to-digital (A/D) conversion togenerate an image signal as digital data.

The volatile memory 160 is composed of, for example, a random accessmemory (RAM), and is used to temporarily hold data. The volatile memory160 is used as a memory for various control operations by themicrocomputer 120, image processing by the image processing unit 190,and the like.

The non-volatile memory 170 is composed of, for example, a read-onlymemory (ROM). The non-volatile memory 170 stores various programs forcausing the microcomputer 120 to operate, correction information for usein the image processing unit 190, and the like.

The microcomputer 120 is a control unit configured to control theoverall operation of the image pickup apparatus 1, such as control ofthe entire image pickup apparatus 1 and image processing sequencecontrol, by using the volatile memory 160 as a work memory, based onprograms stored in the non-volatile memory 170. The microcomputer 120can receive lens information related to the lens apparatus 3 and arrayinformation related to the optical apparatus 2 from the opticalapparatus 2. In other words, the microcomputer 120 functions as a lensinformation acquisition unit and an array information acquisition unitaccording to the disclosure.

The correction information acquisition unit 180 acquires correctioninformation to be used for aberration correction and density correctionon the image signal. The correction information is preliminarilycalculated for each combination of lens information and arrayinformation by the microcomputer 120. The preliminarily calculatedcorrection information is associated with the lens information and thearray information and is stored as a data table. The data table isdesirably stored in the non-volatile memory 170. The correctioninformation acquisition unit 180 compares the acquired lens informationand array information with the data table, whereby correctioninformation to be used for aberration correction and density correctioncan be acquired.

The image processing unit 190 is an image processing unit that performsprocessing, such as processing of cutting out a plurality of images in atile form (hereinafter simply referred to as tile images), aberrationcorrection, and density correction on the image signal output from theimage pickup unit 110. The image processing unit 190 can include adedicated circuit block for specific image processing. The microcomputer120 can perform image processing based on a program.

The operation unit 130 is an operation unit including buttons, switches,dials, and a touch panel that can be manually operated by a user. In theimage pickup apparatus 1, the operation unit 130 receives an operationfrom the user and, based on the operation contents, the microcomputer120 controls each unit of the image pickup apparatus 1 to implement theoperation.

The display unit 140 displays images, a graphical user interface (GUI)screen that is included in a GUI, and the like. The microcomputer 120generates a display control signal based on a program and controls eachunit of the image pickup apparatus 1 to generate a video signal to bedisplayed on the display unit 140 and output the generated video signalto the display unit 140. The components for the display control in theimage pickup apparatus 1 can be up to the interface for outputting thevideo signal to be displayed on the display unit 140, and the displayunit 140 can be an external monitor. The storage unit 150 stores imagedata output from the image processing unit 190. The storage unit 150 canbe incorporated in the image pickup apparatus 1, or can be detachablyattached to the image pickup apparatus 1. The above description detailsthe image pickup apparatus 1 according to the first exemplaryembodiment.

(Configuration of Optical Apparatus 2 and Details of Image PickupSystem)

The optical apparatus 2 and an image pickup system according to thefirst exemplary embodiment will be described below with reference toFIGS. 2A to 2C. FIGS. 2A to 2C are diagrams each illustrating the imagepickup system according to the first exemplary embodiment. FIG. 2Aillustrates a transition of a light beam to be formed in an object imagein the image pickup system using the optical apparatus 2 andtransmission and reception of information between the apparatuses. FIG.2B illustrates a configuration example of a filter array 212 included inthe optical apparatus 2.

As illustrated in FIG. 2A, the image pickup system according to thepresent exemplary embodiment includes the image pickup apparatus 1, theoptical apparatus 2, and the lens apparatus 3 that are arranged in thisorder from an image side. The lens apparatus 3 functions as an opticalapparatus for converting a field angle (image-pickup field angle) of theimage pickup system. A processing unit 310 is a processing unit thatholds lens information related to the lens apparatus 3. Examples of thelens information include an identifier (ID) for identifying a lens typeor an individual lens, information inherent in a lens, such asaberration information, and image pickup information, such as a zoomingposition during an image pickup operation.

An image pickup lens 320 is an optical unit that guides the light beamfrom the object toward the optical apparatus 2 and the image pickupapparatus 1. Although not illustrated in FIG. 2A, the image pickup lens320 includes various lens units, such as a zoom lens, a focus lens, anda shift lens.

The optical apparatus 2 functions as an apparatus for dispersing thelight beam incident from the object through the lens apparatus 3 into aplurality of spectral components. The optical apparatus 2 includes alens array 211 and the filter array 212. The lens array 211 includes aplurality of lens units, each of which forms an image of the object. Thefilter array 212 includes a plurality of filters each arranged on anoptical axis of a different lens unit among the lens unit.

The filter array 212 includes three or more filters arranged in a firstdirection vertical to an optical axis AX0 of the lens apparatus 3 andthe optical apparatus 2. In the present exemplary embodiment, asillustrated in FIG. 2B, the filter array 212 includes nine filters F11to F33 that are arranged in an X-direction and a Y-direction. The filterarray 212 includes a plurality of filters each having a differenttransmission characteristic. This configuration makes it possible tosimultaneously acquire images based on the light beam that is from theobject and is dispersed into a plurality of different spectralcomponents for the same object.

The optical apparatus 2 according to the present exemplary embodimenthas a configuration in which an accessory apparatus 210, including thelens array 211 and the filter array 212, is detachably attachable to theoptical apparatus 2. Specifically, the optical apparatus 2 has anopening (not illustrated) formed on a side portion and the accessoryapparatus 210 is removably insertable into the opening. With thisconfiguration, the filter array 212 is exchangeable to obtain adifferent transmission characteristic as needed, depending on the objectto be picked up or the purpose for image pickup. In the opticalapparatus 2, the lens array 211 is also exchangeable in accordance withthe filter array 212. Accordingly, the number of bands and resolutioncan be adjusted by increasing or decreasing the number of lenses.

A first processing unit 213 is a processing unit that holds arrayinformation related to the optical apparatus 2. Examples of the arrayinformation include an ID for identifying an accessory apparatus type oran individual accessory apparatus, optical characteristics of the lensarray 211, transmission characteristics of the filter array 212, andinformation related to a layout of filters. The optical apparatus 2identifies the type of the accessory apparatus 210, and theabove-described array information is changed as needed.

A second processing unit 220 is a processing unit that transmits thelens information received from the lens apparatus 3 and the arrayinformation acquired from the first processing unit 213 to the imagepickup apparatus 1. As described above, the optical apparatus 2 and theimage pickup apparatus 1 can be coupled together via the mount portion(not illustrated) and exchange various information in communication viarespective communication units, such as electric contacts, each providedon the corresponding mount portion.

The above-described configuration enables the image pickup apparatus 1to detect whether the lens apparatus 3 and the optical apparatus 2 areattached to the image pickup apparatus 1, and to recognize the type ofeach apparatus. Since the image pickup apparatus 1 controls the overalloperation of the image pickup system, establishment of the communicationbetween the optical apparatus 2 and the image pickup apparatus 1 isdesirably performed according to a communication protocol of the imagepickup apparatus 1.

FIG. 2C illustrates areas of the image pickup unit 110 each of whichreceives the light beam from the object via the filter array 212. Asillustrated in FIG. 2C, the image pickup apparatus 1 uses the imagepickup unit 110 (image sensor) to receive the light beam that passedthrough the filter array 212. The light beam passed through each filterof the filter array 212 is received on a different partial area(hereinafter referred to as a tile) among partial areas on the imagesensor corresponding to an arrangement of the filters of the filterarray 212. The image pickup unit 110 (image sensor) outputs an imagesignal (spectrum data) to which an intensity value of the spectralcomponent for the corresponding filter (hereinafter referred to as aspectral intensity) is added. FIG. 2C illustrates nine tiles T11 to T33that are arranged in the X-direction and the Y-direction, which areperpendicular to each other, on the image pickup unit 110 (imagesensor). For example, a light beam that has passed through the filterF11 of the filter array 212 is received by the tile T11.

Specific examples of spectrum data include data on spectroscopicspectrum in an ultraviolet, visible, or infrared region, Ramanspectroscopic spectrum data, nuclear magnetic resonance (NMR) spectrumdata, mass spectrum data, liquid chromatography data, gas chromatographydata, and sound frequency spectrum data. In particular, spectrum datadesirably include any one of data on spectroscopic spectrum in anultraviolet, visible, or infrared region, Raman spectroscopic spectrumdata, and mass spectrum data. In a case where data on spectroscopicspectrum in an ultrasonic, visible, or infrared region or Ramanspectroscopic spectrum data is used as spectrum data, the spectralcomponents can be converted into a wavelength or a wave number. In acase where mass spectrum data is used as spectrum data, the spectralcomponents can be converted into a mass-to-charge ratio or a massnumber. The above description details the optical apparatus 2 and theimage pickup system according to the first exemplary embodiment.

(Details of Aberration Correction Processing)

In the image signal acquired by the image pickup system illustrated inFIG. 2A, an image distortion (distortion) due to a lens or the like, oran image size difference between tile images (magnification chromaticaberration) may occur. An image processing method for reducing thedistortion and the magnification chromatic aberration will be describedbelow. FIG. 3 is a flowchart illustrating aberration correctionprocessing according to the first exemplary embodiment. The imageprocessing method according to the present exemplary embodiment will bedescribed below with reference to FIG. 3 .

In step S301, the microcomputer 120 turns on the power of the imagepickup apparatus 1 in response to a user operation on a power switchincluded in the operation unit 130.

Next, in step S302, the microcomputer 120 acquires lens information fromthe lens apparatus 3 currently attached to the image pickup apparatus 1.As described above, the microcomputer 120 receives the ID or the likefor identifying the type of the attached lens as lens information fromthe lens apparatus 3 via the optical apparatus 2. Prior to executing theprocessing of step S302, the microcomputer 120 determines whether theoptical apparatus 2 and the lens apparatus 3 are attached to the imagepickup apparatus 1 and determines the type of each apparatus and thelike via the electric contacts provided on the mount portions (notillustrated).

Next, in step S303, the microcomputer 120 acquires array informationfrom the optical apparatus 2. As described above, the microcomputer 120receives the ID or the like for identifying the type of each of the lensarray 211 and filter array 212 as the array information.

Next, in step S304, the microcomputer 120 acquires an image signal byusing the image pickup unit 110 picking up an image of the object.

Next, in step S305, the microcomputer 120 acquires information (zoominformation), about the focal length of the lens apparatus 3 and thelike, obtained when the image of the object is picked up in theprocessing of step S304 from the lens apparatus 3 via the opticalapparatus 2.

Next, in step S306, the microcomputer 120 acquires information to beused for various aberration correction operations, based on the acquiredlens information and array information. In general, in a case ofperforming aberration correction processing, a transformation fortransforming coordinates of a pixel in an ideal grid after correctionand coordinates of the corresponding pixel before the correction isused. For example, the coordinates of a pixel P in an ideal grid arerepresented by (x, y), the coordinates of the corresponding pixel P′before the correction are represented by (x′, y′), and transformcoefficients are represented by A0 to A9 and B0 to B9. In this case, forexample, the following transformation (1) can be used. The transformcoefficients A0 to A9 and B0 to B9 are coefficients that vary inaccordance with the lens array 211 and the filter array 212, in additionto the lens type and the zooming position. In particular, as for thefilter array 212, the transform coefficients varies also due to thebandwidth of each band-pass filter and the layout of filters.Accordingly, the microcomputer 120 calculates the transform coefficientsA0 to A9 and B0 to B9 in advance for each combination of the acquiredlens information and array information, and associates the calculatedtransform coefficients with the lens information and array informationto prepare an aberration correction information table. This aberrationcorrection information table is desirably stored in the non-volatilememory 170 of the image pickup apparatus 1, but instead can be held inthe first processing unit 213 of the optical apparatus 2 or theprocessing unit 310 of the lens apparatus 3. In the processing of stepS306, a lens identifier and an array identifier are searched in theaberration correction information table, to acquire the transformcoefficients A0 to A9 and B0 to B9, which are to be used for aberrationcorrection, for each tile image.

$\begin{matrix} & (1)\end{matrix}$ $\begin{pmatrix}x^{\prime} \\y^{\prime}\end{pmatrix} = \begin{pmatrix}{{A0} + {A1x} + {A2y} + {A3x^{2}} + {A4xy} + {A5y^{2}} + {A6x^{3}} + {A7x^{2}y} + {A8xy^{2}} + {A9y^{3}}} \\{{B0} + {B1x} + {B2y} + {B3x^{2}} + {B4xy} + {B5y^{2}} + {B6x^{3}} + {B7x^{2}y} + {B8xy^{2}} + {B9y^{3}}}\end{pmatrix}$

In the case of acquiring the transform coefficients A0 to A9 and B0 toB9 in advance, the transform coefficients are generally calculated basedon the result of picking up an image of a known object, such as apredetermined chart. In this case, the transform coefficients A0 to A9and B0 to B9 are calculated for each tile image in a manner such thatthe predetermined object is located at substantially the same positionwith substantially the same shape and magnification in all tile imagesobtained by picking up an image of the predetermined chart. With thecalculated transform coefficients, the distortion, magnificationchromatic aberration, and positional deviation can be collectivelycorrected.

Next, in step S307, the microcomputer 120 cuts out tile images from theimage signal. The processing of steps S307 and S308 will be described indetail below with reference to FIGS. 4A to 4D. FIGS. 4A to 4D eachillustrate a method for cutting out tile images and performingaberration correction processing according to the first exemplaryembodiment. FIG. 4A illustrates an original image signal. FIG. 4Billustrates tile images before aberration correction. FIG. 4Cillustrates tile images after aberration correction. FIG. 4D illustratestransform coefficients for tile images.

The original image signal illustrated in FIG. 4A indicates an imagesignal based on a light beam received from an object via the nine tilesarranged in the X-direction and Y-direction on the image sensorillustrated in FIG. 2C. The processing of step S307 corresponds toprocessing of cutting out a number of tile images corresponding to thenumber of tiles as illustrated in FIG. 4B from the original image signalillustrated in FIG. 4A.

Next, in step S308, the microcomputer 120 performs aberration correctionprocessing on each tile image, based on the aberration correctioninformation acquired in step S306. Specifically, in step S308, themicrocomputer 120 performs, on all pixels a coordinate transformationbased on the transform coefficients for the tile images acquired in stepS306 and the transformation (1) as illustrated in FIG. 4D andinterpolation processing for determining the spectral intensity afterthe transformation. Application of the above-described processing by themicrocomputer 120 to all tile images generates all tile images afteraberration correction as illustrated in FIG. 4C. In other words, themicrocomputer 120 functions as an image generation unit according to thedisclosure. After the processing of step S308, a plurality of tileimages obtained after aberration correction may be reconfigured togenerate a single multiband image.

Lastly, in step S309, the microcomputer 120 determines whether apower-off instruction is issued by a user operation or the like. In acase where the microcomputer 120 determines that the power-offinstruction is issued (YES in step S309), the aberration correctionprocessing is terminated. In a case where the microcomputer 120determines that the power-off instruction is not issued (NO in stepS309), the processing of steps S304 to S309 is repeated. In a case ofacquiring a moving image, the processing of steps S304 to S309 isrepeated. In a case of acquiring a still image, processing ofdetermining whether a still image pickup instruction is issued can beperformed before step S304 and the processing of steps S304 to S309 canbe repeated based on the result of the determination processing. Theabove description details the aberration correction processing accordingto the present exemplary embodiment.

As described above, the image pickup apparatus 1 according to thepresent exemplary embodiment acquires aberration correction informationcorresponding to each of a plurality of tile images (multiband images),based on lens information and array information, whereby the aberrationcorrection processing can be collectively performed based on theaberration correction information. This configuration eliminates theneed for, for example, executing aberration correction processing oneach of a plurality of images obtained via a filter array including aplurality of filters configured to disperse a light beam into differentspectral components, which leads to a reduction in time required foraberration correction.

Second Exemplary Embodiment

Next, a configuration example of the image pickup apparatus 1 accordingto a second exemplary embodiment of the disclosure will be described.The configurations of the image pickup apparatus 1, the opticalapparatus 2, and the lens apparatus 3 are similar to those of the firstexemplary embodiment described above, and thus the redundantdescriptions are omitted. Only components that are different from thecomponents of the first exemplary embodiment described above will bedescribed.

Even in a case where spectral characteristics of an object in tileimages obtained via the optical apparatus 2 are uniform regardless of awavelength, unevenness in brightness density (unevenness in spectralintensity) can occur due to optical characteristics of the lensapparatus 3, transmission characteristics of the filter array 212 in theoptical apparatus 2, sensitivity characteristics of the image sensor,and the like. FIG. 5 is a flowchart illustrating density correctionprocessing according to the second exemplary embodiment. An imageprocessing method for reducing the effects of density unevenness will bedescribed below with reference to FIG. 5 .

In step S501, the microcomputer 120 turns on the power of the imagepickup apparatus 1 in response to a user operation on the power switchincluded in the operation unit 130. Next, in step S502, themicrocomputer 120 acquires array information from the optical apparatus2. Next, in step S503, the microcomputer 120 acquires an image signal bypicking up an image of the object using the image pickup unit 110.

The processing of steps S501 to S503 is similar to the processing ofsteps S301, S303, and S304 in the first exemplary embodiment describedabove.

Next, in step S504, the microcomputer 120 acquires information ondensity correction processing to be applied to the image signal, basedon the acquired array information. As described above, a difference inbrightness between tile images (density unevenness) can occur due tooptical characteristics of the lens apparatus 3, transmissioncharacteristics of the lens array 211 and the filter array 212 includedin the optical apparatus 2, sensitivity characteristics of the imagepickup unit 110, and the like. The difference in brightness between aplurality of tile images acquired at the same timing may lead to failurein obtaining an accurate processing result in a case of using theplurality of tile images for various processing operations (e.g., whenthe tile images are used as teacher data in artificial intelligence (AI)processing).

Thus, according to the present exemplary embodiment, in the processingof step S504, a density ratio between tile images is calculated inadvance for each combination of the array information and the imagepickup apparatus 1, and a density correction information table isprepared in advance as information related to the array information andthe image pickup apparatus 1 which are associated with each other. Themicrocomputer 120 searches the density correction information table forthe array identifier, to acquire the density ratio between tile imagesto be used for density correction.

This configuration enables the image pickup apparatus 1 according to thepresent exemplary embodiment to correct the brightness (spectralintensity) of each pixel in each tile image, based on the densitycorrection information table every time an image of the object is pickedup, whereby the difference in brightness (density unevenness) betweenthe tile images can be reduced. The density correction information tableis desirably stored in the non-volatile memory 170 of the image pickupapparatus 1, but instead can be held in the first processing unit 213 ofthe optical apparatus 2.

In a case where the density ratio between tile images is preliminarilycalculated, the density ratio is generally calculated based on theresult of picking up an image of a white object, such as a white balancechart. As a method for calculating the density ratio between tile imagesusing such a white balance chart, calculation of a mean value ofspectrum intensities for each tile image is performed for all tileimages generated by picking up an image of the white balance chart. Inthis calculation, it is desirable to calculate the mean value ofspectrum intensities at a central portion of each tile image, to reducethe effects of shading. The mean value of spectrum intensities is set asthe density of each tile image. The spectral intensity can be set as theluminance of each image, and the mean value at a central portion of eachimage can be set as the density of each tile image. Then, a referencetile image is selected and the density ratio between the reference tileimage and the other tile images is calculated, whereby the density ratiobetween tile images is generated.

Table 1 to be described below illustrates an example of the densityratio (brightness ratio) between tile images as density correctioninformation. Table 1 illustrates a case where an array structureincludes a lens array and a filter array each of which are divided into4×3 areas (divided into 12 tiles), and band-pass filters are formed in amanner such that a center wavelengths of the band-pass filters are setin increments of 50 nm in a range from 475 nm to 1025 nm. In the exampleillustrated in Table 1, the tile image corresponding to the band-passfilter with a center wavelength of 975 nm is set as a reference tileimage (density ratio “1”) and the density ratio between the referencetile image and the other tile images is calculated.

TABLE 1 Tile Center Wavelength of Density Ratio Position Band-passFilter [nm] between Tile Images T1l 575 0.30 T12 875 0.72 T13 1025 0.34T14 675 0.57 T21 775 0.84 T22 975 1.00 T23 475 0.16 T24 825 0.63 T31 6250.42 T32 525 0.25 T33 925 0.96 T34 725 0.49

Referring again to FIG. 5 , in step S505, the microcomputer 120 cuts outtile images from the image signal. A method for cutting out tile imagesis substantially the same as step S307 in the first exemplary embodimentdescribed above, and thus the description thereof is omitted.

Next, in step S506, the microcomputer 120 corrects the unevenness indensity between tile images, based on information related to the densityratio acquired in step S504. When the spectral intensity of the pixel atthe position represented by the coordinates (x, y) in a tile image “t”is represented by It(x, y), the spectral intensity obtained afterdensity correction is represented by I′t(x, y), and the density ratiobetween a reference tile image “s” and the tile image “t” is representedby Rst, the following formula (2) can be used as a correction formula.In the processing of step S506, spectral intensity correction processingbased on this correction formula is performed on all pixels in each tileimage, to generate the tile image after density correction. Further, thespectral intensity correction processing is performed on all tileimages, to generate all tile images after density correction.

$\begin{matrix}{{I^{\prime}{t\left( {x,y} \right)}} = \frac{{It}\left( {x,y} \right)}{Rst}} & (2)\end{matrix}$

After the processing of step S506, a plurality of tile images obtainedafter density correction can be reconfigured to generate a singlemultiband image. Lastly, in step S507, the microcomputer 120 determineswhether the power-off instruction is issued by a user operation or thelike. In a case where the microcomputer 120 determines that thepower-off instruction is issued (YES in step S507), the densitycorrection processing is terminated. In a case where the microcomputer120 determines that the power-off instruction is not issued (NO in stepS507), the processing of steps S503 to S507 is repeated. In this case,the processing procedure is substantially similar to the aberrationcorrection processing in the first exemplary embodiment described above,and thus the redundant descriptions are omitted. The above descriptiondetails the density correction processing according to the presentexemplary embodiment.

While the exemplary embodiments of the disclosure are described above,the disclosure is not limited to these exemplary embodiments. Variousmodifications and changes can be made within the scope of thedisclosure. While the above-described exemplary embodiments illustratean example where aberration correction processing and density correctionprocessing are implemented by performing processing in each functionalunit of the image pickup apparatus 1, the disclosure is not limited tothis example. For example, some of the functions of the image pickupapparatus 1 can be implemented by an image processing apparatusconnected with the image pickup apparatus 1 via a network or the like.

While the above-described exemplary embodiments illustrate an examplewhere each of the lens array 211 and the filter array 212 included inthe optical apparatus 2 are divided into nine areas (3×3 areas), thedisclosure is not limited to this example. For example, any number ofareas, such as 4×4 areas or 4×3 areas, can be set and any other settingmethod can be used at least in a case where an aberration correctiontable and a density correction table are calculated for each dividedarea.

While the above-described exemplary embodiments illustrate a method foracquiring lens information and array information from the opticalapparatus 2 and the lens apparatus 3, the disclosure is not limited tothis example. For example, lens information and array information can beacquired from an external apparatus (a personal computer (PC), asmartphone, etc.) that is connectable with the image pickup apparatus 1,and the aberration correction table and the density correction table canbe calculated based on the acquired information. In other words, anyrelevant information can be acquired from an apparatus other than theapparatuses directly connected with the image pickup apparatus 1, andthe aberration correction table and the density correction can becalculated based on the acquired information.

While the above-described exemplary embodiments illustrate an example inwhich a digital camera is used as the image pickup apparatus 1, toimplement the disclosure, the disclosure is not limited to this example.For example, image pickup apparatuses other than a digital camera, suchas a digital video camera, a wearable terminal, and a security cameracan be used.

While the above-described exemplary embodiments illustrate an example inwhich a lens-interchangeable digital camera to which the lens apparatus3 is detachably attached is used as an example of the image pickupapparatus 1, to implement the disclosure, a lens-integrated digitalcamera can also be used. In this case, the digital camera at leasthaving a configuration in which a member substantially the same as theoptical apparatus 2 can be inserted to a main body of the image pickupapparatus 1 or into an optical path on the lens side can be used.

While the above-described exemplary embodiments illustrate aconfiguration in which the overall operation of the image pickupapparatus 1 is controlled by causing the units constituting the imagepickup system to operate in cooperation, based on the microcomputer 120,the disclosure is not limited to this configuration. For example,(computer) programs based on processing procedures illustrated in FIGS.3 and 5 can be preliminarily stored in a ROM area or the like of thenon-volatile memory 170 in the image pickup apparatus 1. Themicrocomputer 120 can execute the programs to control the overalloperation of the image pickup system. A program format to be used is notparticularly limited at least in a case where a program function isprovided. For example, an object code, a program to be executed by aninterpreter, or script data to be supplied to an operating system (OS)can be used. Examples of a recording medium to supply the programs caninclude a hard disk, a magnetic recording medium, such as a magnetictape, and an optical/magneto-optical recording medium.

Other Exemplary Embodiments

The disclosure can also be implemented by processing in which a programfor implementing one or more functions according to the above-describedexemplary embodiments is supplied to a system or an apparatus via anetwork or a storage medium, and one or more processors in the system orthe apparatus read out and execute the program. The disclosure can alsobe implemented by a circuit (e.g., an ASIC) for implementing one or morefunctions according to the above-described exemplary embodiments.

According to an aspect of the disclosure, it is possible to preventdeterioration in image quality due to aberration in a camera systemusing an optical apparatus that is replaceable with respect to an imagepickup apparatus as a multispectral camera.

Other Embodiments

Embodiment(s) of the disclosure can also be realized by a computer of asystem or apparatus that reads out and executes computer executableinstructions (e.g., one or more programs) recorded on a storage medium(which may also be referred to more fully as a ‘non-transitorycomputer-readable storage medium’) to perform the functions of one ormore of the above-described embodiment(s) and/or that includes one ormore circuits (e.g., application specific integrated circuit (ASIC)) forperforming the functions of one or more of the above-describedembodiment(s), and by a method performed by the computer of the systemor apparatus by, for example, reading out and executing the computerexecutable instructions from the storage medium to perform the functionsof one or more of the above-described embodiment(s) and/or controllingthe one or more circuits to perform the functions of one or more of theabove-described embodiment(s). The computer may comprise one or moreprocessors (e.g., central processing unit (CPU), micro processing unit(MPU)) and may include a network of separate computers or separateprocessors to read out and execute the computer executable instructions.The computer executable instructions may be provided to the computer,for example, from a network or the storage medium. The storage mediummay include, for example, one or more of a hard disk, a random-accessmemory (RAM), a read only memory (ROM), a storage of distributedcomputing systems, an optical disk (such as a compact disc (CD), digitalversatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, amemory card, and the like.

While the disclosure has been described with reference to exemplaryembodiments, it is to be understood that the disclosure is not limitedto the disclosed exemplary embodiments. The scope of the followingclaims is to be accorded the broadest interpretation so as to encompassall such modifications and equivalent structures and functions.

This application claims the benefit of Japanese Patent Application No.2021-197123, filed Dec. 3, 2021, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An apparatus comprising: a sensor; at least oneprocessor; and at least one memory coupled to the at least oneprocessor, the at least one memory having instructions that, whenexecuted by the at least one processor, perform to: acquire lensinformation related to a lens apparatus; acquire array informationrelated to an optical apparatus; acquire aberration correctioninformation, based on the lens information and the array information;cut out a plurality of images in a tile form from an image signal outputby the sensor picking up a light beam incident from an object through afilter of the optical apparatus; and correct an aberration in theplurality of images, based on the aberration correction information,wherein the filter includes a plurality of areas to disperse the lightbeam from the object into a plurality of spectral components, andwherein each of the plurality of images corresponds to a different areaamong the plurality of areas of the filter.
 2. The apparatus accordingto claim 1, wherein the array information includes information relatedto a type of the filter.
 3. The apparatus according to claim 1, whereinthe array information includes information related to transmissioncharacteristics of a filter array of the filter and a layout of theplurality of areas.
 4. The apparatus according to claim 1, wherein thefilter is removably insertable into the optical apparatus.
 5. Theapparatus according to claim 1, wherein the filter is disposed in theoptical apparatus.
 6. The apparatus according to claim 1, wherein theoptical apparatus includes a lens array including a plurality of lensunits, and wherein each of the plurality of images corresponds to adifferent lens unit among the plurality of lens units in the lens array.7. The apparatus according to claim 6, wherein the array informationincludes information related to a type of the lens array.
 8. Theapparatus according to claim 6, wherein the array information includesinformation related to optical characteristics of the lens array and alayout of the plurality of lens units.
 9. The apparatus according toclaim 6, wherein the lens array is removably insertable into the opticalapparatus.
 10. The apparatus according to claim 6, wherein the lensarray is disposed in the optical apparatus.
 11. The apparatus accordingto claim 1, wherein the lens information includes information related toa type of the lens apparatus.
 12. The apparatus according to claim 1,wherein the lens information includes information related to a zoomingposition.
 13. The apparatus according to claim 1, wherein the aberrationcorrection information is calculated for correction of the plurality ofimages acquired by picking up of a predetermined object, and by usingthe aberration correction information, a size, and a magnification ofthe predetermined object will be same between the plurality of images.14. The apparatus according to claim 13, wherein the aberrationcorrection information is used to correct a pixel position in each ofthe plurality of images.
 15. An optical apparatus comprising: a sensor;at least one processor; and at least one memory coupled to the at leastone processor, the at least one memory having instructions that, whenexecuted by the at least one processor, perform to: acquire lensinformation related to a lens apparatus; acquire array informationrelated to the optical apparatus; acquire aberration correctioninformation, based on the lens information and the array information;cut out a plurality of images in a tile form from an image signal outputby the sensor picking up a light beam incident from an object through afilter of the optical apparatus; and correct an aberration in theplurality of images, based on the aberration correction information,wherein the filter includes a plurality of areas to disperse the lightbeam from the object into a plurality of spectral components, andwherein each of the plurality of images corresponds to a different areaamong the plurality of areas of the filter.
 16. The optical apparatusaccording to claim 15, wherein the optical apparatus is detachablyattachable to a mount portion provided on the apparatus.
 17. Anapparatus comprising: at least one processor; and at least one memorycoupled to the at least one processor, the at least one memory havinginstructions that, when executed by the at least one processor, performto: acquire lens information related to a lens apparatus; acquireaberration correction information, based on the lens information andarray information related to an optical apparatus; and transmit thecorrection information to the image pickup apparatus, wherein apredetermined filter includes a plurality of areas to disperse a lightbeam from an object into a plurality of spectral components.
 18. Amethod to control an apparatus comprising a sensor, the methodcomprising: acquiring lens information related to a lens apparatus;acquiring array information related to an optical apparatus; acquiringaberration correction information, based on the lens information and thearray information; cutting out a plurality of images in a tile form froman image signal output by the sensor picking up a light beam incidentfrom an object through a filter of the optical apparatus; and correctingan aberration in the plurality of images, based on the aberrationcorrection information, wherein the filter includes a plurality of areasto disperse the light beam from the object into a plurality of spectralcomponents.
 19. A non-transitory computer-readable storage medium whichstores a program for causing a computer of an apparatus to execute amethod to control the apparatus comprising a sensor and being able to beconnected with a lens apparatus and an optical apparatus, the methodcomprising: acquiring lens information related to the lens apparatus;acquiring array information related to the optical apparatus; acquiringaberration correction information, based on the lens information and thearray information; cutting out a plurality of images in a tile form froman image signal output by the sensor picking up a light beam incidentfrom an object through a filter of the optical apparatus; and correctingan aberration in the plurality of images, based on the aberrationcorrection information, wherein the filter includes a plurality of areasto disperse the light beam from the object into a plurality of spectralcomponents.